{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Importance of Feature Scaling\n\n\nFeature scaling through standardization (or Z-score normalization)\ncan be an important preprocessing step for many machine learning\nalgorithms. Standardization involves rescaling the features such\nthat they have the properties of a standard normal distribution\nwith a mean of zero and a standard deviation of one.\n\nWhile many algorithms (such as SVM, K-nearest neighbors, and logistic\nregression) require features to be normalized, intuitively we can\nthink of Principle Component Analysis (PCA) as being a prime example\nof when normalization is important. In PCA we are interested in the\ncomponents that maximize the variance. If one component (e.g. human\nheight) varies less than another (e.g. weight) because of their\nrespective scales (meters vs. kilos), PCA might determine that the\ndirection of maximal variance more closely corresponds with the\n'weight' axis, if those features are not scaled. As a change in\nheight of one meter can be considered much more important than the\nchange in weight of one kilogram, this is clearly incorrect.\n\nTo illustrate this, PCA is performed comparing the use of data with\n:class:`StandardScaler <sklearn.preprocessing.StandardScaler>` applied,\nto unscaled data. The results are visualized and a clear difference noted.\nThe 1st principal component in the unscaled set can be seen. It can be seen\nthat feature #13 dominates the direction, being a whole two orders of\nmagnitude above the other features. This is contrasted when observing\nthe principal component for the scaled version of the data. In the scaled\nversion, the orders of magnitude are roughly the same across all the features.\n\nThe dataset used is the Wine Dataset available at UCI. This dataset\nhas continuous features that are heterogeneous in scale due to differing\nproperties that they measure (i.e alcohol content, and malic acid).\n\nThe transformed data is then used to train a naive Bayes classifier, and a\nclear difference in prediction accuracies is observed wherein the dataset\nwhich is scaled before PCA vastly outperforms the unscaled version.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import PCA\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn import metrics\nimport matplotlib.pyplot as plt\nfrom sklearn.datasets import load_wine\nfrom sklearn.pipeline import make_pipeline\nprint(__doc__)\n\n# Code source: Tyler Lanigan <tylerlanigan@gmail.com>\n#              Sebastian Raschka <mail@sebastianraschka.com>\n\n# License: BSD 3 clause\n\nRANDOM_STATE = 42\nFIG_SIZE = (10, 7)\n\n\nfeatures, target = load_wine(return_X_y=True)\n\n# Make a train/test split using 30% test size\nX_train, X_test, y_train, y_test = train_test_split(features, target,\n                                                    test_size=0.30,\n                                                    random_state=RANDOM_STATE)\n\n# Fit to data and predict using pipelined GNB and PCA.\nunscaled_clf = make_pipeline(PCA(n_components=2), GaussianNB())\nunscaled_clf.fit(X_train, y_train)\npred_test = unscaled_clf.predict(X_test)\n\n# Fit to data and predict using pipelined scaling, GNB and PCA.\nstd_clf = make_pipeline(StandardScaler(), PCA(n_components=2), GaussianNB())\nstd_clf.fit(X_train, y_train)\npred_test_std = std_clf.predict(X_test)\n\n# Show prediction accuracies in scaled and unscaled data.\nprint('\\nPrediction accuracy for the normal test dataset with PCA')\nprint('{:.2%}\\n'.format(metrics.accuracy_score(y_test, pred_test)))\n\nprint('\\nPrediction accuracy for the standardized test dataset with PCA')\nprint('{:.2%}\\n'.format(metrics.accuracy_score(y_test, pred_test_std)))\n\n# Extract PCA from pipeline\npca = unscaled_clf.named_steps['pca']\npca_std = std_clf.named_steps['pca']\n\n# Show first principal components\nprint('\\nPC 1 without scaling:\\n', pca.components_[0])\nprint('\\nPC 1 with scaling:\\n', pca_std.components_[0])\n\n# Use PCA without and with scale on X_train data for visualization.\nX_train_transformed = pca.transform(X_train)\nscaler = std_clf.named_steps['standardscaler']\nX_train_std_transformed = pca_std.transform(scaler.transform(X_train))\n\n# visualize standardized vs. untouched dataset with PCA performed\nfig, (ax1, ax2) = plt.subplots(ncols=2, figsize=FIG_SIZE)\n\n\nfor l, c, m in zip(range(0, 3), ('blue', 'red', 'green'), ('^', 's', 'o')):\n    ax1.scatter(X_train_transformed[y_train == l, 0],\n                X_train_transformed[y_train == l, 1],\n                color=c,\n                label='class %s' % l,\n                alpha=0.5,\n                marker=m\n                )\n\nfor l, c, m in zip(range(0, 3), ('blue', 'red', 'green'), ('^', 's', 'o')):\n    ax2.scatter(X_train_std_transformed[y_train == l, 0],\n                X_train_std_transformed[y_train == l, 1],\n                color=c,\n                label='class %s' % l,\n                alpha=0.5,\n                marker=m\n                )\n\nax1.set_title('Training dataset after PCA')\nax2.set_title('Standardized training dataset after PCA')\n\nfor ax in (ax1, ax2):\n    ax.set_xlabel('1st principal component')\n    ax.set_ylabel('2nd principal component')\n    ax.legend(loc='upper right')\n    ax.grid()\n\nplt.tight_layout()\n\nplt.show()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}